ZMQ¶

Contexts¶

In ZeroMQ proper, a context is an opaque, thread-safe handle to manage ØMQ's resources within a process. Typically, a process will allocate only one context, although more than one context per process is allowable [ref].

In Chapel, a Context is a reference-counted wrapper around the underlying ØMQ context. To create a context, it is sufficient to write:

var context: Context;

Sockets¶

In ZeroMQ, a socket is an opaque handle to an asynchronous, message-based communication channel that is "typed" to provide one of a series of common communication patterns (i.e., the socket type).

In Chapel, as with a Context, a Socket is a reference-counted wrapper around the underlying ØMQ socket. Sockets are created via Context.socket() and maintain a reference to the parent context, so that the parent context may go out of scope and the context will not be reclaimed while any sockets are still in use.

A Socket may be one of the socket types in the following list of compatible pairs of socket types [ref]:

• PUB and SUB
• REQ and REP
• PUSH and PULL

// create a PUB socket
var context: Context;
var socket = context.socket(ZMQ.PUB);

Sending and Receiving Messages¶

Sending a message is as simple as passing the object to send as the argument to Socket.send(). Receiving a message requires that the type to be received be passed as the argument to Socket.recv(). In either case, if the object sent or type to be received cannot be serialized by ZMQ, the following error shall be produced at compile time.

// send or receive an int
var val = 42;
socket.send(val);
val = socket.recv(int);

// error: Type "Foo" is not serializable by ZMQ
class Foo { var val: int; }
socket.recv(Foo);

Multilocale Support¶

Chapel's ZMQ module supports multilocale execution of ZeroMQ programs. The locale on which the Context object is created sets the "home" locale for all sockets created from this context and all operations performed on the socket. For example, a send call on a socket from a locale other than the home locale will migrate a task to the home locale that will remotely access the data to send it over the socket.

Example: "Hello, World"¶

This "Hello, World" example demonstrates a PUSH-PULL socket pair in two Chapel programs that exchange a single string message.

// pusher.chpl
use ZMQ;
config const to: string = "world!";
var context: Context;
var socket = context.socket(ZMQ.PUSH);
socket.bind("tcp://*:5555");
socket.send(to);

// puller.chpl
use ZMQ;
var context: Context;
var socket = context.socket(ZMQ.PULL);
socket.connect("tcp://localhost:5555");
writeln("Hello, ", socket.recv(string));

Serialization¶

In Chapel, sending or receiving messages is supported for a variety of types. Primitive numeric types and strings are supported as the foundation. In addition, user-defined record types may be serialized automatically as multipart messages by internal use of the Reflection module. Currently, the ZMQ module can serialize records of primitive numeric types, strings, and other serializable records.

Interoperability¶

The ZeroMQ messaging library has robust support in many programming languages and Chapel's ZMQ module intends on providing interfaces and serialization protocols suitable for exchanging data between Chapel and non-Chapel programs.

As an example (and test) of Chapel-Python interoperability over ZeroMQ, the following sources demonstrate a REQ-REP socket pair between a Chapel server and a Python client using the PyZMQ Python bindings for ZeroMQ.

  begin {
    var client = spawn(["./wrapper.sh", "client.py"]);
    client.communicate();
  }

// server.chpl
use ZMQ;

var context: Context;
var socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");

for i in 0..#10 {
  var msg = socket.recv(string);
  writeln("[Chapel] Received message: ", msg);
  socket.send("Hello %i from Chapel".format(i));
}

# client.py
import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

for request in range(10):
    message = "Hello %i from Python" % request
    print("[Python] Sending request: %s" % message)
    socket.send_string(message)
    message = socket.recv_string()
    print("[Python] Received response: %s" % message)

Tasking-Layer Interaction¶

As noted previously, a Socket object is not thread safe. As the mapping of Chapel tasks to OS threads is dependent on the Chapel tasking layer and may be cooperatively scheduled, a Socket object should not be accessed concurrently by multiple Chapel tasks.

The ZMQ module is designed to "play nicely" with the Chapel tasking layer. While the C-level call zmq_send() may be a blocking call (depending on the socket type and flag arguments), it is desirable that a semantically-blocking call to Socket.send() allow other Chapel tasks to be scheduled on the OS thread as supported by the tasking layer. Internally, the ZMQ module uses non-blocking calls to zmq_send() and zmq_recv() to transfer data, and yields to the tasking layer via chpl_task_yield() when the call would otherwise block.

Limitations and Future Work¶

Currently, the ZMQ module does not provide interfaces for working with ZeroMQ message objects, handling errors, or making explicitly non-blocking send/receive calls. These are active-but-incomplete areas of work that are intended to be supported in future Chapel releases.

One interaction of these features is worth noting explicitly: because multipart messages are used to automatically serialize non-primitive data types (e.g., strings and records) and a partially-sent multi-part message cannot be canceled (except by closing the socket), an explicitly non-blocking send call that encountered an error in the ZeroMQ library during serialization would not be in a recoverable state, nor would there be a matching "partial receive".