5 Distributed programming with CapTP

CAUTION: CapTP support is still in early days. Things will change, and some things (such as sturdyrefs and "three-machine introductions") are missing. Please do not rely yet on this protocol being stable. Also be aware that the interface provided in this release is known to be suboptimally "manual"; in future releases the system will help users set up CapTP connections more easily.

CapTP is a system for enabling distributed programming in a mutually suspicious network. It originated in the E programming language but has since appeared in Cap’N Proto, Waterken, Agoric’s SwingSet, and now Goblins.

5.1 What CapTP gives you

• Invocation of remote objects resembles the same as with local objects. You can use <- and on which means that any most programs that were even originally designed mostly for local-only computing naturally scale out to a networked environment.

• All this is done while mostly hiding the abstraction of the networked protocol from the user.

• Object capability security is upheld. A remote machine cannot make use of any capability that has not been handed to it.

• Live object references between CapTP endpoints are incredibly cheap, merely represented as integers on each side. This keeps message sizes small and efficient.

• CapTP provides distributed garbage collection. Remote machines can indicate when they no longer need an object and the machine locally containing that reference can reclaim it if appropriate.

• Promise pipelining means that messages can be sent to the resolution of a promise before that resolution actually occurs. Over the network this is represented as a pipeline of messages. This can reduce round trips significantly, which is a big win. To re-quote Mark S. Miller:

  "Machines grow faster and memories grow larger. But the speed of light is constant and New York is not getting any closer to Tokyo."

• CapTP is written independently of network transport abstractions. It can be run over local unix domain sockets, an OpenSSL connection, Tor Onion Services, or something custom. CapTP operates under the assumption of secure pairwise channels; the layer which provides the pairwise channels is called "MachineTP" and can be written in a variety of ways. A simple MachineTP abstraction is provided, but presently still requires some manual wiring. (In the future, even simpler abstractions will be provided which cover most user needs.)

All of these combine into an efficient and powerful protocol that, most powerfully of all, the author of Goblins programs mostly need not think too much about. This circumvents the usual years and years of careful message structure coordination and standardization for many kinds of distributed programs, which can simply be written as normal Goblins programs rather than bespoke protocols.

5.2 CapTP Limitations

• Goblins’ CapTP provides acyclic distributed garbage collection. Cycles between servers are not automatically recognized. Full cycle-collecting distributed garbage collection has been written, but requires special cooperation from the garbage collector that we don’t have access to in Racket (or in most languages).

• Goblins’ CapTP does not do anything about memory usage or resource management on its own. (Features for this will come as Spritely sub-projects in the future.)

• While CapTP in theory routes capabilities to specific remote objects, since the network is mutually suspicious we can’t assume that the remote end isn’t conspiring some way to hand those capabilities to other objects that we don’t expect. The right way to think about this from an object capability perspective is that a remote misbehaving machine is equivalent to a misbehaving object with the surface area of the entire machine.

• TEMPORARY: We haven’t exposed easy bootstrapping of CapTP connections. Currently you have to manually wire them, which is a pain. In the future we will have some kind of "sturdyref" type feature that will make it easier (through certificate chains or URIs with bearer tokens) to set up new connections.

• TEMPORARY: Multi-machine introductions have not yet been written. If Alice, Bob, and Carol are all on three different machines and Alice would like to hand Bob a reference to Carol, there is no good way to do that presently. This will be fixed in the future in such a way that developers need not think about / be aware of this.

5.3 CapTP usage example

As said in CapTP Limitations, we don’t currently have an "easy" way to connect together CapTP connections. For the moment you have to manually wire them together. In the future this will be fixed.

In the meanwhile, here’s a simple version. What we are going to do is write two programs which talk over local "unix domain sockets". However you could just as easily use an OpenSSL connection or Tor Onion Services or whatever you choose.

Here’s our server, save as captp-uds-server.rkt:

#lang racket

(require goblins

goblins/captp

racket/unix-socket)

;; The ^car-factory example from the promise pipelining part

;; of the Goblins tutorial

(define (^car-factory bcom company-name)

(define ((^car bcom model color))

(format "*Vroom vroom!*  You drive your ~a ~a ~a!"

color company-name model))

(define (make-car model color)

(spawn ^car model color))

make-car)

;; Handle a new incoming connection on the unix domain socket

(define (handle-uds-listen uds-listener bootstrap-actor)

;; Accept the connection, getting a new input and output port

(define-values (ip op)

(unix-socket-accept uds-listener))

;; Set up a machinetp thread using this input and output port pair

(define _remote-bootstrap-vow

(make-machinetp-thread ip op

;; the bootstrap actor is whatever we

;; passed in (in this case, the car factory)

bootstrap-actor))

(displayln "*** New connection!"))

(define (run-server uds-path)

;; A vat to spawn some objects in

(define car-vat

(make-vat))

(define fork-motors

(car-vat 'spawn ^car-factory "Fork"))

;; Listen to the unix domain socket at uds-path

(define uds-listener

(unix-socket-listen uds-path))

(displayln "*** Server up and running!")

(dynamic-wind

void

(lambda ()

(let lp ()

(sync (handle-evt uds-listener

(lambda _

(handle-uds-listen uds-listener fork-motors))))

(lp)))

(lambda ()

(delete-file uds-path))))

(module+ main

(command-line

#:args (uds-path)

(run-server uds-path)))

Now save the following client code as captp-uds-client.rkt:

#lang racket

(require goblins

goblins/captp

goblins/actor-lib/bootstrap

racket/unix-socket)

(define (run-client uds-path signal-when-done)

;; We'll use this vat to run some code and set up our bootstrap actor

(define a-vat

(make-vat))

;; Here's where we'll connect to the unix domain path

(define-values (ip op)

(unix-socket-connect uds-path))

(define remote-bootstrap-vow

(make-machinetp-thread ip op))

;; Well we're being incredibly lazy and the remote bootstrap actor

;; *is* the car factory, so let's just alias that

(define car-factory-vow remote-bootstrap-vow)

(displayln "*** Connected to server")

(define (get-and-drive-car)

;; Get a promise to get a new car

(define car-vow

(<- car-factory-vow "Explorist" "blue"))

;; Pipeline a promise with the noise from driving it

;; (The car takes no arguments on invocation to drive)

(define drive-noise-vow

(<- car-vow))

;; Now listen to the resolution of the pipelined driving noise.

(on drive-noise-vow

(lambda (drive-noise)

;; Yay, promise pipelining worked!

(displayln (format "We hear: ~a" drive-noise))

;; Okay now signal that we can shut this down

(semaphore-post signal-when-done))

#:catch

(lambda (err)

(displayln (format "UHOH!!! Something went wrong: ~a" err)))))

;; Now run the above thunk on a-vat

(a-vat 'run get-and-drive-car))

(module+ main

(define signal-when-done

(make-semaphore))

(command-line

#:args (uds-path)

(run-client uds-path signal-when-done))

;; Keep this open until we've finished our communication, since the

;; rest of the program runs in a thread.

(semaphore-wait signal-when-done))

In general when connecting to a captp connection without prior introductions in the network, you need some way of getting capabilities. In many cases this will be a generalized "sturdyref" provider (either use a random nonce or invoke a certificate chain to look up the first capability you’ve been authorized access to... see spawn-nonce-registry-locator-pair as one example of how to do this) but we’re hand-waving past that for the sake of minimalism here. Instead we’re just slotting the car factory itself as the capability the user is connecting to.

Okay, let’s try it out. Open up two terminals, and in the first one run:

And in the second run:

You should see:

If that worked, your connection succeeded, and so did promise pipelining... horray!

5.4 CapTP API

Returns a promise for the bootstrap object on the other end.

captp-vat is a vat connector (such as one returned from make-vat), which is where the various actors which manage this captp connection’s state will live. send-to-remote is a procedure which takes a Syrup record which it will send across some connection. The remote side will be provided bootstrap-obj as the resolution of its bootstrap promise.

Returns two values to its continuation, the first being a reference to an actor which receives messages from the machinetp connection, and the second being a promise to the remote object’s promise resolver.

With this first returned value and the send-to-remote procedure, we effectively have a way to receive and send messages over captp and we could map this on top of a network layer or something else. This is exactly what make-machinetp-thread does with ports.