5 Distributed programming with CapTP

CAUTION: CapTP support is still in early days. Please do not rely yet on this protocol being stable.

CapTP is a system for enabling distributed programming in a mutually suspicious network. It originated in the E programming language but has since appeared in Cap’N Proto, Waterken, Agoric’s SwingSet, and now Goblins.

5.1 What CapTP gives you

• Invocation of remote objects resembles the same as with local objects. You can use <- and on which means that any most programs that were even originally designed mostly for local-only computing naturally scale out to a networked environment.

• All this is done while mostly hiding the abstraction of the networked protocol from the user.

• Object capability security is upheld. A remote machine cannot make use of any capability that has not been handed to it.

• Live object references between CapTP endpoints are incredibly cheap, merely represented as integers on each side. This keeps message sizes small and efficient.

• CapTP provides (acyclic) distributed garbage collection. Remote machines can indicate when they no longer need an object and the machine locally containing that reference can reclaim it if appropriate.

• Promise pipelining means that messages can be sent to the resolution of a promise before that resolution actually occurs. Over the network this is represented as a pipeline of messages. This can reduce round trips significantly, which is a big win. To re-quote Mark S. Miller:

  "Machines grow faster and memories grow larger. But the speed of light is constant and New York is not getting any closer to Tokyo."

• CapTP is written independently of network transport abstractions. It can be run over local unix domain sockets, an OpenSSL connection, Tor Onion Services, or something custom. CapTP operates under the assumption of secure pairwise channels; the layer which provides the pairwise channels is called "MachineTP" and can be written in a variety of ways. A simple MachineTP abstraction is provided, but presently still requires some manual wiring. (In the future, even simpler abstractions will be provided which cover most user needs.)

All of these combine into an efficient and powerful protocol that, most powerfully of all, the author of Goblins programs mostly need not think too much about. This circumvents the usual years and years of careful message structure coordination and standardization for many kinds of distributed programs, which can simply be written as normal Goblins programs rather than bespoke protocols.

5.2 CapTP Limitations

• Goblins’ CapTP provides acyclic distributed garbage collection. Cycles between servers are not automatically recognized. Full cycle-collecting distributed garbage collection has been written, but requires special cooperation from the garbage collector that we don’t have access to in Racket (or in most languages).

• Goblins’ CapTP does not do anything about memory usage or resource management on its own. (Features for this will come as Spritely sub-projects in the future.)

• While CapTP in theory routes capabilities to specific remote objects, since the network is mutually suspicious we can’t assume that the remote end isn’t conspiring some way to hand those capabilities to other objects that we don’t expect. The right way to think about this from an object capability perspective is that a remote misbehaving machine is equivalent to a misbehaving object with the surface area of the entire machine.

• TEMPORARY: The API is heavily in flux.

5.3 CapTP usage example

There’s still work to be done in terms of making Goblins’ CapTP "easier" to use and set up. In the meanwhile, let’s give an example showing off Goblins’ Tor Onion Services support.

5.3.1 Launch a Tor daemon for Goblins

In order for this to work, Goblins needs a Tor daemon to talk to... Goblins can set up and manage the onion services itself once it has that. So far it seems like running a single Goblins tor process "as your user" is the easiest option, and let various Goblins processes connect to that.

Here’s an example config file template... replace <user> with your actual username:

Put in ~/.config/goblins/tor-config.txt or something.

You’ll also want to make the relevant directories:

Okay, now you can launch the Tor daemon:

In the future we should probably make a raco launcher that does the above for users so they don’t need to read this section. :P

5.3.2 Two Goblins programs chatting over CapTP

Here’s our server, save as captp-alice.rkt:

#lang racket

(require goblins

goblins/actor-lib/bootstrap

goblins/ocapn

goblins/ocapn/netlayer/onion

net/url)

;; Machine to talk to the network on

(define-vat-run machine-run (make-vat))

(define mycapn

(machine-run

(spawn-mycapn (setup-onion-netlayer))))

;; Get the command line argument for Alice's address

(define alice-sref

(command-line #:args (alice-sref-str)

(string->ocapn-sturdyref alice-sref-str)))

;; Set up a vat for "alice"

(define-vat-run a-run (make-vat))

(a-run

(define alice-vow (<- mycapn 'enliven alice-sref))

(on (<- alice-vow "Alice")

(lambda (heard-back)

(displayln (format "Alice heard back: ~a" heard-back))

;; we're done

(semaphore-post finished))))

;; A semaphore for us to signal once we've heard back from Alice

(define finished (make-semaphore))

(sync finished)

Traditionally Alice talks to Bob, you see. So now we need Bob.

Now save the following client code as captp-bob.rkt:

#lang racket

(require goblins

goblins/actor-lib/bootstrap

goblins/ocapn

goblins/ocapn/netlayer/onion

net/url)

;; Bob is going to be a "greeter" of sorts, so let's define what

;; that is

(define (^greeter _bcom my-name)   ; constructor args

(lambda (your-name)              ; invocation args

(format "Hello ~a, my name is ~a!" your-name my-name)))

;; Now let's make a vat for bob, and run bob in it

(define-vat-run b-run (make-vat))

(define bob (b-run (spawn ^greeter "Bob")))

;; First, let's set up our "machine" for the network

(define-vat-run machine-run (make-vat))

(machine-run

(define mycapn

(spawn-mycapn (setup-onion-netlayer)))

;; Now we need a sturdyref for Bob... let's print it out so that

;; we know how to connect

(define bob-sturdyref

($ mycapn 'register bob 'onion))

;; And let's print out to the command line

(displayln (format "Bob's sturdyref: ~a"

(url->string (ocapn-sturdyref->url bob-sturdyref)))))

;; And we need some way to keep things open despite the other vats

;; humming along in their own threads... this kluge does it

(sync (make-semaphore))

Okay, let’s try it out. Open up two terminals, and in the first one run:

It’ll spit out a captp "sturdyref" URI. Copy it and in a new terminal paste it as the argument to captp-alice.rkt:

You should see:

If that worked, your connection succeeded! Horray!

5.4 CapTP API

Returns an object representing the machine with various methods that Chris really ought to document.

Sorry, these docs are pretty sparse; we hope to make them better. Hopefully the above example helps get you started.