ZMQ¶
Usage
use ZMQ;
Lightweight messaging with ZeroMQ (or ØMQ)
This module provides high-level Chapel bindings to the ZeroMQ messaging library.
Dependencies¶
The ZMQ module in Chapel is dependent on ZeroMQ. For information on how to install ZeroMQ, see the ZeroMQ installation instructions.
Note
Chapel's support for ZeroMQ is a work in progress and may not (yet) support the full functionality of the ZeroMQ C API.
Note
Chapel's ZMQ module was developed for compatibility with ZeroMQ v4.x.
Using ZMQ in Chapel¶
Contexts¶
In ZeroMQ proper, a context is an opaque, thread-safe handle to manage ØMQ's resources within a process. Typically, a process will allocate only one context, although more than one context per process is allowable [ref].
In Chapel, a Context
is a reference-counted wrapper around the
underlying ØMQ context. To create a context, it is sufficient to write:
var context: Context;
Sockets¶
In ZeroMQ, a socket is an opaque handle to an asynchronous, message-based communication channel that is "typed" to provide one of a series of common communication patterns (i.e., the socket type).
In Chapel, as with a Context
, a Socket
is a
reference-counted wrapper around the underlying ØMQ socket. Sockets are
created via Context.socket()
and maintain a reference to the parent
context, so that the parent context may go out of scope and the context will
not be reclaimed while any sockets are still in use.
Note
As with ØMQ's C API, a Socket
object is not thread safe.
That is, a Socket
object should not be accessed concurrently by
multiple Chapel tasks. (This may change in a future ZMQ module.)
A Socket
may be one of the socket types in the following list of
compatible pairs of socket types
[ref]:
// create a PUB socket
var context: Context;
var socket = context.socket(ZMQ.PUB);
Sending and Receiving Messages¶
Sending a message is as simple as passing the object to send as the argument
to Socket.send()
. Receiving a message requires that the type to be
received be passed as the argument to Socket.recv()
.
In either case, if the object sent or type to be received cannot be serialized
by ZMQ, the following error shall be produced at compile time.
// send or receive an int
var val = 42;
socket.send(val);
val = socket.recv(int);
// error: Type "Foo" is not serializable by ZMQ
class Foo { var val: int; }
socket.recv(Foo);
Multilocale Support¶
Chapel's ZMQ module supports multilocale execution of ZeroMQ programs. The
locale on which the Context
object is created sets the "home" locale
for all sockets created from this context and all operations performed on the
socket. For example, a send call on a socket from a locale other than the
home locale will migrate a task to the home locale that will remotely access
the data to send it over the socket.
Examples¶
Example: "Hello, World"¶
This "Hello, World" example demonstrates a PUSH
-PULL
socket
pair in two Chapel programs that exchange a single string message.
// pusher.chpl
use ZMQ;
config const to: string = "world!";
var context: Context;
var socket = context.socket(ZMQ.PUSH);
socket.bind("tcp://*:5555");
socket.send(to);
// puller.chpl
use ZMQ;
var context: Context;
var socket = context.socket(ZMQ.PULL);
socket.connect("tcp://localhost:5555");
writeln("Hello, ", socket.recv(string));
Implementation Notes¶
As noted previously, the ZMQ module is a work in progress. The implementation notes below describe some of how the ZMQ module is implemented and how future versions may expose more features native to ZeroMQ.
Serialization¶
In Chapel, sending or receiving messages is supported for a variety of types.
Primitive numeric types and strings are supported as the foundation.
In addition, user-defined record
types may be serialized automatically
as multipart messages
by internal use of the Reflection
module.
Currently, the ZMQ module can serialize records of primitive numeric types,
strings, and other serializable records.
Note
The serialization protocol for strings changed in Chapel 1.16 in order to support inter-language messaging through ZeroMQ. (See notes on interoperability below.) As a result, programs using ZMQ that were compiled by Chapel 1.15 or earlier cannot communicate using strings with programs compiled by Chapel 1.16 or later, although such communication can be used between programs compiled by the same version without issue.
Prior to Chapel 1.16, ZMQ would send a string as two multipart messages: the first sent the length as int; the second sent the character array (in bytes). It was identified that this scheme was incompatible with how other language bindings for ZeroMQ serialize and send strings.
As of Chapel 1.16, the ZMQ module uses the C-level zmq_msg_send()
and
zmq_msg_recv()
API for Socket.send()
and Socket.recv()
,
respectively, when transmitting strings. Further, ZMQ sends the string as
a single message of only the byte stream of the string's character array.
(Recall that Chapel's string
type currently only supports ASCII
strings, not full Unicode strings.)
Interoperability¶
The ZeroMQ messaging library has robust support in many programming languages and Chapel's ZMQ module intends on providing interfaces and serialization protocols suitable for exchanging data between Chapel and non-Chapel programs.
As an example (and test) of Chapel-Python interoperability over ZeroMQ, the
following sources demonstrate a PUSH
-PULL
socket pair between
a Chapel server and a Python client using the
PyZMQ Python bindings for ZeroMQ.
// server.chpl
use ZMQ;
var context: Context;
var socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");
for i in 0..#10 {
var msg = socket.recv(string);
writeln("[Chapel] Received message: ", msg);
socket.send("Hello %i from Chapel".format(i));
}
# client.py
import zmq
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
for request in range(10):
message = "Hello %i from Python" % request
print("[Python] Sending request: %s" % message)
socket.send_string(message)
message = socket.recv_string()
print("[Python] Received response: %s" % message)
Tasking-Layer Interaction¶
As noted previously, a Socket
object is not thread safe. As the
mapping of Chapel tasks to OS threads is dependent on the Chapel tasking layer
and may be cooperatively scheduled, a Socket
object should not be
accessed concurrently by multiple Chapel tasks.
The ZMQ module is designed to "play nicely" with the Chapel tasking layer.
While the C-level call zmq_send()
may be a blocking call (depending on
the socket type and flag arguments), it is desirable that a
semantically-blocking call to Socket.send()
allow other Chapel tasks
to be scheduled on the OS thread as supported by the tasking layer.
Internally, the ZMQ module uses non-blocking calls to zmq_send()
and
zmq_recv()
to transfer data, and yields to the tasking layer via
chpl_task_yield() when the call would otherwise block.
Limitations and Future Work¶
Currently, the ZMQ module does not provide interfaces for working with ZeroMQ message objects, handling errors, or making explicitly non-blocking send/receive calls. These are active-but-incomplete areas of work that are intended to be supported in future Chapel releases.
One interaction of these features is worth noting explicitly: because multipart messages are used to automatically serialize non-primitive data types (e.g., strings and records) and a partially-sent multi-part message cannot be canceled (except by closing the socket), an explicitly non-blocking send call that encountered an error in the ZeroMQ library during serialization would not be in a recoverable state, nor would there be a matching "partial receive".
References¶
-
const
PUB
= ZMQ_PUB¶ The publisher socket type for a publish-subscribe messaging pattern.
-
const
SUB
= ZMQ_SUB¶ The subscriber socket type for a publish-subscribe messaging pattern.
-
const
REQ
= ZMQ_REQ¶ The requester socket type for a paired request-reply messaging pattern.
-
const
REP
= ZMQ_REP¶ The replier socket type for a paired request-reply messaging pattern.
-
const
PUSH
= ZMQ_PUSH¶ The pusher socket type for a pipeline messaging pattern.
-
const
PULL
= ZMQ_PULL¶ The puller socket type for a pipeline messaging pattern.
-
const
SUBSCRIBE
= ZMQ_SUBSCRIBE¶ The
Socket.setsockopt()
option value to specify the message filter for aSUB
-typeSocket
.
-
const
UNSUBSCRIBE
= ZMQ_UNSUBSCRIBE¶ The
Socket.setsockopt()
option value to remote an existing message filter for aSUB
-typeSocket
.
-
const
LINGER
= ZMQ_LINGER¶ The
Socket.setsockopt()
option value to specify the linger period for the associatedSocket
object.
-
proc
version
: (int, int, int)¶ Query the ZMQ library version.
Returns: An (int,int,int) tuple of the major, minor, and patch version of the ZMQ library.
-
record
Context
¶ A ZeroMQ context. See more on using Contexts. Note that this record contains private fields not listed below.
-
proc
init
()¶ Create a ZMQ context.
-
proc
-
record
Socket
¶ A ZeroMQ socket. See more on using Sockets. Note that this record contains private fields not listed below.
-
proc
close
(linger: int = unset)¶ Close the socket.
Arguments: linger : int -- Optional argument to specify the linger period for the socket prior to closing. If -1, then the linger period is infinite; if non-negative, then the linger period shall be set to the specified value (in milliseconds).
-
proc
bind
(endpoint: string)¶ Bind the socket to the specified local endpoint and accept incoming connections.
-
proc
connect
(endpoint: string)¶ Connect the socket to the specified endpoint.
-
proc
setsockopt
(option: int, value: ?T)¶ Set socket options; see zmq_setsockopt
Arguments: - option : int -- a socket option;
e.g.,
LINGER
,SUBSCRIBE
,UNSUBSCRIBE
- value -- the socket option value
- option : int -- a socket option;
e.g.,
-
proc
send
(data: ?T)¶ Send an object data on a socket.
Arguments: data -- The object to be sent. If data is an object whose type is not serializable by the ZMQ module, a compile-time error will be raised.
-
proc
recv
(type T): T¶ Receive an object of type T from a socket.
Arguments: T -- The type of the object to be received. If T is not serializable by the ZMQ module, a compile-time error will be raised. Returns: An object of type T
-
proc